Falco: Runtime Threat Detection with eBPF

Threat detection has lived through two eras: perimeter (firewalls, network IDS) and endpoint (EDR on laptops). Containers and Kubernetes break both assumptions: perimeter is blurry, endpoints are ephemeral. Falco, a graduated CNCF project, attacks the problem from a different direction: watch the kernel and alert when something does unexpected things.

This article covers what Falco does well, what it doesn’t do, and what it needs around it to be operationally useful.

What Falco Watches

Falco hooks the Linux kernel (via eBPF or a traditional module) and observes syscalls in real time. Each syscall — open, execve, connect, setuid — passes through its rules. If one fires, Falco emits an event.

Examples of default rules:

Shell in container: an interactive shell opens inside a production container.
Write to /etc: a process modifies sensitive files.
Binary executed from /tmp: classic dropper pattern.
Unexpected outbound connection: container calls an unknown IP.
Privilege change: a process escalates to root without it being its usual CMD.

The power is in granularity: not “the container did something bad”, but “PID 1234 called execve with argv=/bin/sh inside this pod’s namespace, and that doesn’t match baseline”.

eBPF vs Kernel Module Architecture

Falco supports two drivers:

Modern eBPF (CO-RE, since Falco 0.34+): portable across kernels, no compilation, no loaded module. The recommended one.
Legacy eBPF: requires kernel headers at build time.
Kernel module: oldest. Still useful on kernels without eBPF CO-RE, but increasingly irrelevant.

For Debian 12+, Ubuntu 22.04+, Amazon Linux 2023, RHEL 9+ the modern eBPF driver works without fighting. On very old kernels (<5.8) you may need the module.

Rules: The 80/20

Falco ships with falco_rules.yaml and ~80 out-of-the-box rules. The reasonable path:

Start with defaults. See what noise your real environment generates in the first 2-3 days.
Silence common false positives. Your backup copies files to /etc/cron.d — looks like “write to sensitive path” but is legitimate.
Add stack-specific rules (e.g. “nobody runs kubectl exec outside office hours”).
Invest the rest in runbooks for real events.

Rules useful beyond defaults:

- rule: AWS credentials access
  desc: Access to .aws directory or EC2 metadata
  condition: (fd.name in (~/.aws/credentials) or fd.name=169.254.169.254)
  output: "Possible AWS credential extraction (user=%user.name)"
  priority: WARNING

Rules are in a custom DSL — simple but powerful. The official tutorial covers essentials.

Output: SIEM and Notification

Falco emits JSON events. What you do with them defines utility:

Falcosidekick (official companion): event router to Slack, PagerDuty, Elasticsearch, Loki, Alertmanager, and dozens more.
Loki + Grafana: events as logs, alerts via Grafana Alerting. Good for teams already on the stack.
SIEM (Splunk, Sentinel, Elastic Security): Falco as one detection source among several.

The common error: leave events in /var/log/falco.log and never look. Without routing to a reviewed destination, Falco is local noise.

Deployment on Kubernetes

In K8s, Falco runs as a DaemonSet — one pod per node, with privileged access to the node’s kernel.

Official Helm chart: falcosecurity/falco. Options for driver, rules, event export.
Requires privileged securityContext or hostPID + hostNetwork + specific capabilities. That’s risk surface — Falco has more privileges than almost anything in the cluster.
Resources: ~200MB RAM, 0.1-0.5 CPU per node at rest. Spikes during mass pod starts.
Frequent updates: threat rules change; keep Falco current.

Where Falco Falls Short

Being honest about limits:

Memory-only attacks (no observable syscalls) fly under the radar.
Malicious behaviour inside normal syscalls (e.g. exfil via normal HTTPS to a C2) requires destination-specific rules.
Sophisticated cryptojacking can avoid obvious patterns.
Kernel zero-days that subvert eBPF.

Falco is not antivirus nor WAF. It’s anomaly detection on syscalls. Complements other layers; doesn’t replace them.

Kubernetes Audit Integration

Beyond syscalls, Falco can consume the Kubernetes API audit log and apply rules over API actions (e.g. “someone created a Secret with a suspicious name”). Covers the control-plane layer syscalls don’t touch.

Useful for detecting:

RBAC changes outside hours.
Mass creation of privileged pods.
Unusual Secrets access.

Operations: What Hurts

A year running Falco in production teaches:

Updating rules without versioning is chaotic. Keep rulesets in Git with CI validating them.
Drift between Falco and kernel. Each major kernel update can break eBPF CO-RE via edge cases.
Alert fatigue. If 90% of alerts are benign, the important 10% is lost. Invest in refinement.
Cost on dense nodes. 200 pods/node = many syscalls. Monitor Falco CPU.
Team turnover. If nobody understands the rules, nobody refines. Document.

When to Adopt Falco

Clearly useful:

Kubernetes in production with third-party pods or multi-tenant.
Compliance requiring runtime monitoring (PCI DSS, mature SOC 2).
Teams with SOC able to triage events.

Marginal or overkill:

A single Docker node. Falco is too much; a well-configured auditd is simpler.
Teams without response process. Alerts without response are theatre.

Conclusion

Falco is a mature, open-source piece for runtime detection in cloud-native environments. Requires investment in operation — tuned rules, event routing, response — but in return offers visibility traditional layers don’t reach. Before adopting, ask whether your team can process its events; if not, prioritise that before deployment. Detecting without responding is noise; detecting with response is real defence.